Convolutional Neural Networks

Need for Convolution Neural Network

In the previous tutorials we saw learned about the Feed Forward Neural Networks where each neuron in a network learns from every neuron in the previous layer. This architecture of the network assumes independence between the features of the data. This helps in learning the abstract global representation. However, this architure cannot learn local representations of the data because there is no interaction between the neighboring neurons. For Example we are analyzing an image or a video or a spech sequence we need to learn the local representations of the and build on those local representations to form a global representation.

Convolution and Pooling layer

We attain this function by imposing sparsity on the layers such that most of the weight are zeros exept few. We attain sparcity by forcig each neuron to learn only from new neighboring inputs. Let $k$ be number of neighnors we allow a neuron to learn from. If we take $k=3$ then each neuron only get input from the the 3 previous neurons. Forcibly reducing the inputs of the neuron to 3 will allow the neuron to learn the local representations of the image. For an image case that initial convolution layer learns small edges from the image. Another advantage of this technique is good old reusuability. Once is neuron is trained to learn an edge from an image, we can use the same neuron to detect the edge wherever it is present in the image by moving it around the entire image. We can see this in two ways we can think of this as a single neuron moving around the image to detect the features or many neurons sharing the same weights. This operaiton is called convolution.

Let's look at it mathematically. Let $x = [x^{(1)}, x^{(2)},......,x^{(d)}]$ d dimensional input vector and $W = [w_1, w_2,.......,w_3]$ be the weight vector of the neuron. This operation can be mathematically represented as $$z^{(j)} = \sum_{i=1}^{k}x^{j+i-1}W_k, \forall j=[1,2,…,π‘‘βˆ’π‘˜+1] $$

Receptive Field

The value k we used before that represents the number of inputs the neuron learns from can be seen as the number of signals it recieves from the previous layer. Therefore, this k is called the receptive field. It can also be viewed in a different way. If k is 3 then the neurons receptive field is 3 from the previous layer. What about the neuron's receptive field with respect to the layer before that? The previous layer has each neuron recieved from 3 inputs from the layer before. Thefore the current neuron recieves 9 inputs from the 2 layers before.

Pooling Operation

From the previous section we learned that each neuron recieves input from k neighboring k values in the input and produces one ouput. As the neuron moves in the entire layer, it produces d-k+1 outputs where d is the number of dimensions in the input. Each neuron produces d-k+1 oututs, so if we have n neurons in each layer then we have $n*(d-k+1)$ outputs. For Example for a 20 neuron layer, if the input is 100 dimensional, for k=3 we have 1960 outputs. from 100 to 1960 is a huge jump and this keeps on increasing and will result in a computational instability. To prevent that we have a special operation called pooling. We downsample the entire output space into lower dimension by taking a single value for a given pooling length. depending on the value we consider we call it either as avg pooling or max pooling.

Stride

We can reduce the number of outputs by using strides without using pooling. A stride operation is equivalent to a hop. Stride is the number of hops we make from convolution to convolution. We can use longer strides where we don't want outputs to read from common inputs.

Convolution layer in YANN

We can add a convolution layer to our network using add_layer function with the following syntax: net.add_layer ( type = "conv_pool", origin = "input", id = "conv_pool_1", num_neurons = filter_size = (,), pool_size = (,), activation = ('maxout', 'maxout', 2), batch_norm = True, regularize = True, verbose = verbose )


InΒ [8]:
from yann.network import network
from yann.utils.graph import draw_network
from yann.special.datasets import cook_mnist
def lenet5 ( dataset= None, verbose = 1, regularization = None ):             
    """
    This function is a demo example of lenet5 from the infamous paper by Yann LeCun. 
    This is an example code.  
    
    Warning:
        This is not the exact implementation but a modern re-incarnation.

    Args: 
        dataset: Supply a dataset.    
        verbose: Similar to the rest of the dataset.
    """
    optimizer_params =  {        
                "momentum_type"       : 'nesterov',             
                "momentum_params"     : (0.65, 0.97, 30),      
                "optimizer_type"      : 'rmsprop',                
                "id"                  : "main"
                        }

    dataset_params  = {
                            "dataset"   : dataset,
                            "svm"       : False, 
                            "n_classes" : 10,
                            "id"        : 'data'
                      }

    visualizer_params = {
                    "root"       : 'lenet5',
                    "frequency"  : 1,
                    "sample_size": 144,
                    "rgb_filters": True,
                    "debug_functions" : False,
                    "debug_layers": False,  # Since we are on steroids this time, print everything.
                    "id"         : 'main'
                        }       

    # intitialize the network
    net = network(   borrow = True,
                     verbose = verbose )                       
    
    # or you can add modules after you create the net.
    net.add_module ( type = 'optimizer',
                     params = optimizer_params, 
                     verbose = verbose )

    net.add_module ( type = 'datastream', 
                     params = dataset_params,
                     verbose = verbose )

    net.add_module ( type = 'visualizer',
                     params = visualizer_params,
                     verbose = verbose 
                    )
    # add an input layer 
    net.add_layer ( type = "input",
                    id = "input",
                    verbose = verbose, 
                    datastream_origin = 'data', # if you didnt add a dataset module, now is 
                                                 # the time. 
                    mean_subtract = False )
    
    # add first convolutional layer
    net.add_layer ( type = "conv_pool",
                    origin = "input",
                    id = "conv_pool_1",
                    num_neurons = 20,
                    filter_size = (5,5),
                    pool_size = (3,3),
                    activation = ('maxout', 'maxout', 2),
                    # regularize = True,
                    verbose = verbose
                    )

    net.add_layer ( type = "conv_pool",
                    origin = "conv_pool_1",
                    id = "conv_pool_2",
                    num_neurons = 50,
                    filter_size = (3,3),
                    pool_size = (1,1),
                    activation = 'relu',
                    # regularize = True,
                    verbose = verbose
                    )      


    net.add_layer ( type = "dot_product",
                    origin = "conv_pool_2",
                    id = "dot_product_1",
                    num_neurons = 1250,
                    activation = 'relu',
                    # regularize = True,
                    verbose = verbose
                    )

    net.add_layer ( type = "dot_product",
                    origin = "dot_product_1",
                    id = "dot_product_2",
                    num_neurons = 1250,                    
                    activation = 'relu',  
                    # regularize = True,    
                    verbose = verbose
                    ) 
    
    net.add_layer ( type = "classifier",
                    id = "softmax",
                    origin = "dot_product_2",
                    num_classes = 10,
                    # regularize = True,
                    activation = 'softmax',
                    verbose = verbose
                    )

    net.add_layer ( type = "objective",
                    id = "obj",
                    origin = "softmax",
                    objective = "nll",
                    datastream_origin = 'data', 
                    regularization = regularization,                
                    verbose = verbose
                    )
                    
    learning_rates = (0.05, .0001, 0.001)  
    net.pretty_print()  
    draw_network(net.graph, filename = 'lenet.png')   

    net.cook()

    net.train( epochs = (20, 20), 
               validate_after_epochs = 1,
               training_accuracy = True,
               learning_rates = learning_rates,               
               show_progress = True,
               early_terminate = True,
               patience = 2,
               verbose = verbose)

    print(net.test(verbose = verbose))
data = cook_mnist()
dataset = data.dataset_location()
lenet5 ( dataset, verbose = 2)


. Setting up dataset 
.. setting up skdata
... Importing mnist from skdata
.. setting up dataset
.. training data
.. validation data 
.. testing data 
. Dataset 74549 is created.
. Time taken is 0.480078 seconds
. Initializing the network
.. Setting up the optimizer
.. Setting up the datastream
.. Setting up the visualizer
.. Adding input layer input
.. Adding conv_pool layer conv_pool_1
.. Adding conv_pool layer conv_pool_2
.. Adding dot_product layer dot_product_1
.. Adding flatten layer 4
.. Adding dot_product layer dot_product_2
.. Adding classifier layer softmax
.. Adding objective layer obj
.. This method will be deprecated with the implementation of a visualizer,also this works only for tree-like networks. This will cause errors in printing DAG-style networks.
 |-
 |-
 |-
 |- id: input
 |-=================------------------
 |- type: input
 |- output shape: (500, 1, 28, 28)
 |------------------------------------
          |-
          |-
          |-
          |- id: conv_pool_1
          |-=================------------------
          |- type: conv_pool
          |- output shape: (500, 10, 8, 8)
          |- batch norm is OFF
          |------------------------------------
          |- filter size [5 X 5]
          |- pooling size [3 X 3]
          |- stride size [1 X 1]
          |- input shape [28 28]
          |- input number of feature maps is 1
          |------------------------------------
                   |-
                   |-
                   |-
                   |- id: conv_pool_2
                   |-=================------------------
                   |- type: conv_pool
                   |- output shape: (500, 50, 6, 6)
                   |- batch norm is OFF
                   |------------------------------------
                   |- filter size [3 X 3]
                   |- pooling size [1 X 1]
                   |- stride size [1 X 1]
                   |- input shape [8 8]
                   |- input number of feature maps is 10
                   |------------------------------------
                            |-
                            |-
                            |-
                            |- id: 4
                            |-=================------------------
                            |- type: flatten
                            |- output shape: (500, 1800)
                            |------------------------------------
                                     |-
                                     |-
                                     |-
                                     |- id: dot_product_1
                                     |-=================------------------
                                     |- type: dot_product
                                     |- output shape: (500, 1250)
                                     |- batch norm is OFF
                                     |------------------------------------
                                              |-
                                              |-
                                              |-
                                              |- id: dot_product_2
                                              |-=================------------------
                                              |- type: dot_product
                                              |- output shape: (500, 1250)
                                              |- batch norm is OFF
                                              |------------------------------------
                                                       |-
                                                       |-
                                                       |-
                                                       |- id: softmax
                                                       |-=================------------------
                                                       |- type: classifier
                                                       |- output shape: (500, 10)
                                                       |------------------------------------
                                                                |-
                                                                |-
                                                                |-
                                                                |- id: obj
                                                                |-=================------------------
                                                                |- type: objective
                                                                |- output shape: (1,)
                                                                |------------------------------------
.. Saving the network down as an image
.. Cooking the network
.. Setting up the resultor
.. All checks complete, cooking continues
. Training
. 

.. Epoch: 0 Era: 0
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 76.45
.. Training accuracy : 74.75
.. Best training accuracy
.. Best validation accuracy
.. Cost                : 1.71155
... Learning Rate       : 9.99999974738e-05
... Momentum            : 0.649999976158
. 

.. Epoch: 1 Era: 0
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 88.34
.. Training accuracy : 87.104
.. Best training accuracy
.. Best validation accuracy
.. Cost                : 0.5723
... Learning Rate       : 9.49999957811e-05
... Momentum            : 0.660666584969
. 

.. Epoch: 2 Era: 0
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 92.43
.. Training accuracy : 91.708
.. Best training accuracy
.. Best validation accuracy
.. Cost                : 0.350749
... Learning Rate       : 9.02499959921e-05
... Momentum            : 0.671333312988
. 

.. Epoch: 3 Era: 0
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 94.65
.. Training accuracy : 94.146
.. Best training accuracy
.. Best validation accuracy
.. Cost                : 0.243391
... Learning Rate       : 8.57374980114e-05
... Momentum            : 0.681999981403
. 

.. Epoch: 4 Era: 0
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 95.95
.. Training accuracy : 95.55
.. Best training accuracy
.. Best validation accuracy
.. Cost                : 0.179968
... Learning Rate       : 8.14506202005e-05
... Momentum            : 0.692666649818
. 

.. Epoch: 5 Era: 0
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 96.66
.. Training accuracy : 96.414
.. Best training accuracy
.. Best validation accuracy
.. Cost                : 0.14017
... Learning Rate       : 7.73780921008e-05
... Momentum            : 0.703333318233
. 

.. Epoch: 6 Era: 0
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 97.09
.. Training accuracy : 97.034
.. Best training accuracy
.. Best validation accuracy
.. Cost                : 0.114157
... Learning Rate       : 7.3509188951e-05
... Momentum            : 0.713999986649
. 

.. Epoch: 7 Era: 0
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 97.43
.. Training accuracy : 97.526
.. Best training accuracy
.. Best validation accuracy
.. Cost                : 0.0956501
... Learning Rate       : 6.98337316862e-05
... Momentum            : 0.724666655064
. 

.. Epoch: 8 Era: 0
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 97.65
.. Training accuracy : 97.812
.. Best training accuracy
.. Best validation accuracy
.. Cost                : 0.0818051
... Learning Rate       : 6.63420432829e-05
... Momentum            : 0.735333323479
. 

.. Epoch: 9 Era: 0
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 97.81
.. Training accuracy : 98.078
.. Best training accuracy
.. Best validation accuracy
.. Cost                : 0.0710442
... Learning Rate       : 6.30249414826e-05
... Momentum            : 0.745999991894
. 

.. Epoch: 10 Era: 0
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 97.91
.. Training accuracy : 98.284
.. Best training accuracy
.. Best validation accuracy
.. Cost                : 0.062519
... Learning Rate       : 5.9873695136e-05
... Momentum            : 0.756666600704
. 

.. Epoch: 11 Era: 0
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 97.96
.. Training accuracy : 98.464
.. Best training accuracy
.. Best validation accuracy
.. Cost                : 0.0554453
... Learning Rate       : 5.68800096516e-05
... Momentum            : 0.767333269119
. 

.. Epoch: 12 Era: 0
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 98.05
.. Training accuracy : 98.624
.. Best training accuracy
.. Best validation accuracy
.. Cost                : 0.0495137
... Learning Rate       : 5.40360088053e-05
... Momentum            : 0.777999937534
. 

.. Epoch: 13 Era: 0
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 98.17
.. Training accuracy : 98.714
.. Best training accuracy
.. Best validation accuracy
.. Cost                : 0.0443415
... Learning Rate       : 5.13342092745e-05
... Momentum            : 0.788666665554
. 

.. Epoch: 14 Era: 0
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 98.26
.. Training accuracy : 98.812
.. Best training accuracy
.. Best validation accuracy
.. Cost                : 0.039851
... Learning Rate       : 4.87674988108e-05
... Momentum            : 0.799333274364
. 

.. Epoch: 15 Era: 0
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 98.24
.. Training accuracy : 98.906
.. Best training accuracy
.. Cost                : 0.0360325
... Learning Rate       : 4.63291253254e-05
... Momentum            : 0.80999994278
. 

.. Epoch: 16 Era: 0
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 98.22
.. Training accuracy : 98.966
.. Best training accuracy
.. Cost                : 0.0327061
... Learning Rate       : 4.40126677859e-05
... Momentum            : 0.820666670799
. 

.. Epoch: 17 Era: 0
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 98.17
.. Training accuracy : 98.984
.. Best training accuracy
.. Cost                : 0.0297545
... Learning Rate       : 4.18120325776e-05
... Momentum            : 0.83133327961
. 

.. Epoch: 18 Era: 0
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 98.2
.. Training accuracy : 99.024
.. Best training accuracy
.. Cost                : 0.0271253
... Learning Rate       : 3.97214316763e-05
... Momentum            : 0.841999948025
. 

.. Epoch: 19 Era: 0
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 98.18
.. Training accuracy : 99.07
.. Best training accuracy
.. Cost                : 0.0248562
... Learning Rate       : 3.77353608201e-05
... Momentum            : 0.85266661644
.. Learning rate was already lower than specified. Not changing it.
.. Old learning rate was :3.58485922334e-05
.. Was trying to change to: 0.001
. 

.. Epoch: 20 Era: 1
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 98.19
.. Training accuracy : 98.884
.. Cost                : 0.0368601
... Learning Rate       : 3.58485922334e-05
... Momentum            : 0.863333284855
. 

.. Epoch: 21 Era: 1
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 98.18
.. Training accuracy : 98.934
.. Cost                : 0.0334258
... Learning Rate       : 3.40561637131e-05
... Momentum            : 0.87399995327
. 

.. Epoch: 22 Era: 1
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 98.17
.. Training accuracy : 98.966
.. Cost                : 0.030458
... Learning Rate       : 3.23533568007e-05
... Momentum            : 0.884666621685
. 

.. Epoch: 23 Era: 1
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 98.21
.. Training accuracy : 98.972
.. Cost                : 0.027694
... Learning Rate       : 3.07356895064e-05
... Momentum            : 0.8953332901
. 

.. Epoch: 24 Era: 1
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 98.1
.. Training accuracy : 98.94
.. Cost                : 0.025167
... Learning Rate       : 2.91989053949e-05
... Momentum            : 0.905999958515
. 

.. Epoch: 25 Era: 1
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 98.07
.. Training accuracy : 98.904
.. Cost                : 0.0230637
... Learning Rate       : 2.77389608527e-05
... Momentum            : 0.91666662693
. 

.. Epoch: 26 Era: 1
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 98.1
.. Training accuracy : 98.944
.. Cost                : 0.0214425
... Learning Rate       : 2.63520123553e-05
... Momentum            : 0.927333295345
. 

.. Epoch: 27 Era: 1
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 98.14
.. Training accuracy : 99.096
.. Best training accuracy
.. Cost                : 0.0211709
... Learning Rate       : 2.503441101e-05
... Momentum            : 0.93799996376
. 

.. Epoch: 28 Era: 1
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 98.31
.. Training accuracy : 99.31
.. Best training accuracy
.. Best validation accuracy
.. Cost                : 0.0218001
... Learning Rate       : 2.37826898228e-05
... Momentum            : 0.948666632175
. 

.. Epoch: 29 Era: 1
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 98.3
.. Training accuracy : 99.232
.. Cost                : 0.0222707
... Learning Rate       : 2.25935546041e-05
... Momentum            : 0.959333300591
. 

.. Epoch: 30 Era: 1
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 98.17
.. Training accuracy : 98.94
.. Cost                : 0.0250492
... Learning Rate       : 2.1463876692e-05
... Momentum            : 0.969999969006
. 

.. Epoch: 31 Era: 1
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 98.33
.. Training accuracy : 99.236
.. Best validation accuracy
.. Cost                : 0.0261564
... Learning Rate       : 2.03906820389e-05
... Momentum            : 0.97000002861
. 

.. Epoch: 32 Era: 1
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 98.52
.. Training accuracy : 99.47
.. Best training accuracy
.. Best validation accuracy
.. Cost                : 0.0215522
... Learning Rate       : 1.93711475731e-05
... Momentum            : 0.97000002861
. 

.. Epoch: 33 Era: 1
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 98.48
.. Training accuracy : 99.626
.. Best training accuracy
.. Cost                : 0.0145997
... Learning Rate       : 1.84025902854e-05
... Momentum            : 0.97000002861
. 

.. Epoch: 34 Era: 1
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 98.66
.. Training accuracy : 99.706
.. Best training accuracy
.. Best validation accuracy
.. Cost                : 0.0115174
... Learning Rate       : 1.74824599526e-05
... Momentum            : 0.97000002861
. 

.. Epoch: 35 Era: 1
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 98.63
.. Training accuracy : 99.634
.. Cost                : 0.0104252
... Learning Rate       : 1.66083373188e-05
... Momentum            : 0.97000002861
. 

.. Epoch: 36 Era: 1
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 98.46
.. Training accuracy : 99.512
.. Cost                : 0.0109738
... Learning Rate       : 1.57779195433e-05
... Momentum            : 0.97000002861
. 

.. Epoch: 37 Era: 1
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 98.59
.. Training accuracy : 99.692
.. Cost                : 0.00921145
... Learning Rate       : 1.4989023839e-05
... Momentum            : 0.97000002861
. 

.. Epoch: 38 Era: 1
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 98.54
.. Training accuracy : 99.72
.. Best training accuracy
.. Cost                : 0.00735605
... Learning Rate       : 1.42395729199e-05
... Momentum            : 0.97000002861
. 

.. Epoch: 39 Era: 1
| training  100% Time: 0:00:02                                                 
| validation  100% Time: 0:00:01                                               
.. Validation accuracy : 98.66
.. Training accuracy : 99.79
.. Best training accuracy
.. Cost                : 0.00573931
... Learning Rate       : 1.3527594092e-05
... Momentum            : 0.97000002861
.. Training complete.Took 7.28132636667 minutes
.. Testing
| testing  100% Time: 0:00:00                                                  
.. Testing accuracy : 98.96
None